Using Iperf to Diagnose Network Speed Issues
Many folks are unaware of the dramatic effect disk speeds have on there network speeds, since the bulk of your data comes from or goes to disk, and mass storage is the slowest link in the chain, it is the single most important factor in how fast you can download media home, download torrent payloads, or even upload to others. The chain is: DISK < > Cache/Memory < > Network When tuning a system these are the three components that garner the most attention: how can I get things on to and off of my disk quickly; how best to optimize my cache; and then finally how can I insure my network is running at its peak. In a Reddit post we looked at how best diagnose route and peering issues using the linux command mtr. But mtr looks at road conditions, what is likely slowing you down. It doesn't address how fast you can actually go. We can do that with Iperf, and significantly iperf removes your disk from the mix. MTR lays out the map, the roads you are taking, any potholes, traffic jams, bad interchanges, each turn and curve along the way. Iperf looks at how fast your car can go down that route, pure speed, in real numbers. MTR lays out the track, iperf runs the race. This is a tutorial, it is recommended you follow along using your server and your home machine to run the commands in parallel. Iperf, or its latest incarnation, iperf3, is a client/server tool. You run iperf on your server, then from a client connect and see how fast your server either receives or can send you packets. It is really quite simple. First we need to install iperf3 on your server, as always if you have superuser this is simple, if not a bit harder, with sudo (everything here is debian or ubuntu): sudo apt-get install iperf3 If you don't have root you'll need to manually install it, here is what you need to do: mkdir -p ~/bin ~/lib wget --no-check-certificate -O ~/bin/iperf3 https://iperf.fr/download/ubuntu/iperf3_3.1.3 chmod +x bin/iperf3 export PATH=$PATH:~/bin wget --no-check-certificate -O ~/lib/libiperf.so.0 https://iperf.fr/download/ubuntu/libiperf.so.0_3.1.3 echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/lib" >>~/.bashrc source ~/.bashrc Running things on the client is now easier than tying your shoes: iperf -p 10666 -s 10666 is the port that we will be using, the server will be listen for connections on. It is optional, but since the default port might be taken (5201), I've included how to set your own. Now you need to install the client, a client for virtually any platform can be found over at iperf.fr, in particular https://iperf.fr/iperf-download.php I'm going to assume you have windows, but it is largely the same for any non-Unix client, you download the zip file, unzip it into its own directory, then open a DOS or command window in that directory. With Windows, iperf3 uses the cygwin runtime in the form of DLL(s). Ok, with our server set up, and the client installed, a quick aside for those unanointed in parlance of networking. The two most common Internet protocols are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). TCP is like a phone call, a circuit is created between the caller and the person being called, they say something you'll hear it unless the whole line drops, it is reliable. UDP is more like sending a letter, you drop a packet on the wire with an address and best effort is made to get it delivered, it is inherently unreliable. We'll be using both in our tests. Thing to understand is UDP it is cheaper, and its unreliability can be useful. Lets start with the simplest case: iperf3 -p 10666 -c server.hostname.or.ip This will SEND packets to the server for 10 seconds, and tell you how fast the packets went, the output will look like: E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com Connecting to host iperftest.chmuranet.com, port 10666 [ 4] local 192.168.1.19 port 10358 connected to .... port 10666 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 1.00 MBytes 8.39 Mbits/sec [ 4] 1.00-2.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 2.00-3.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 3.00-4.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 4.00-5.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 5.00-6.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 6.00-7.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 7.00-8.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 8.00-9.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 9.00-10.00 sec 2.25 MBytes 18.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 21.2 MBytes 17.8 Mbits/sec sender [ 4] 0.00-10.00 sec 21.2 MBytes 17.8 Mbits/sec receiver The last set of numbers (sender / receiver) is our summary line, average speed. This is in megabits per second, so my upload speed is 2.23MB/s. This though is just a test of how fast I can talk to my server, which isn't nearly as interesting as how fast my server can talk to me, to do that we need to tell iperf to receive. We add the -R parameter E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com -R [ 4] 0.00-1.00 sec 1.62 MBytes 13.6 Mbits/sec [ 4] 1.00-2.00 sec 1.65 MBytes 13.8 Mbits/sec [ 4] 2.00-3.00 sec 1.07 MBytes 8.99 Mbits/sec [ 4] 3.00-4.00 sec 814 KBytes 6.67 Mbits/sec [ 4] 4.00-5.00 sec 510 KBytes 4.18 Mbits/sec [ 4] 5.00-6.00 sec 446 KBytes 3.66 Mbits/sec [ 4] 6.00-7.00 sec 205 KBytes 1.68 Mbits/sec [ 4] 7.00-8.00 sec 99.8 KBytes 818 Kbits/sec [ 4] 8.00-9.00 sec 141 KBytes 1.16 Mbits/sec [ 4] 9.00-10.00 sec 345 KBytes 2.83 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 8.83 MBytes 7.41 Mbits/sec 22 sender [ 4] 0.00-10.00 sec 6.97 MBytes 5.85 Mbits/sec receiver Now this is interesting, my speed down is hitting Sunday night traffic, and is significantly slower. See the number of retries (Retr)? There is some problem on the down route, you can also see the jitter (variations in latency or speed). Now you should compare this to the speed your are getting on a single FTP connection using like Filezilla. Checking my connection, I'm getting .5MB/s that means the disk is costing me 25% of my download speed, not unusual. If the disk is re To see how fast I can get things to, like segmented FTP, I'm going to need to use multiple simultaneous connections, lucky iperf3 can do that too. Lets do 10 segments, with the -P 10 parameter. E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com -R -P 10 SUM 0.00-10.00 sec 30.2 MBytes 25.3 Mbits/sec 239 sender SUM 0.00-10.00 sec 26.0 MBytes 21.8 Mbits/sec receiver Just the summary lines, but you can see much faster. Lets try 30. SUM 0.00-10.00 sec 77.0 MBytes 64.6 Mbits/sec 947 sender SUM 0.00-10.00 sec 64.4 MBytes 54.0 Mbits/sec receiver That scales rather nicely, I am receiving 6.75MB/s, but look at my retransmits, that has scaled too. Server is sending at over 8MB/s. At this point further escalation of connections, doesn't scale nearly as well. OK, lets see what might be going on, we'll use UDP now (by adding the -u parameter), with 10 connections, from me to my server, : E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com -P 10 -u SUM 0.00-10.00 sec 15.6 MBytes 13.1 Mbits/sec 2.989 ms 0/1998 (0%) No packet loss, of the 1998 packets sent 0% were lost, at a speed of 13.1Mbits/s - pretty good. Now going the other way (-R): E:\mtrs\iperf-3.1.3-win64>iperf3 -p 10666 -c iperftest.chmuranet.com -P 10 -u -R SUM 0.00-10.00 sec 12.7 MBytes 10.6 Mbits/sec 1.192 ms 9/1620 (0.56%) Fewer packets sent and an overall loss of .56% of the packets wandered off. I'm willing to bet if I did an mtr, I'd see some bad jitter, and probably some packet loss. Yep, checking mtr, we see where the problem is: 4. ae-21-52.car1.Chicago1.Level3.net 1.9% 340 95.5 95.6 95.4 97.0 23 1.9% packet loss in Chicago, those ruffians. So you can see, iperf can be an effective tool, primarily for determining the hit you are taking because of your hard disk, but also in analyzing route dynamics, what is costing you against your ideal. I've covered most of the interesting parameters, the only other flag I use is -t as in -t 30, so that iperf will run for longer (30 secs vs 10 sec default), which can give you a better picture. But with all linux commands, you should take a look at the manpage yourself: https://www.mankier.com/1/iperf3 Disclaimer: I fiddled with the numbers to make this more interesting, I also ran a laptop against a Chmuranet 10G server, a DC based CPU-laden 10G server can send packets much faster then my poor laptop can handle, this is part of the packet loss you saw in my examples. Consider this when looking at your own server.